fix(tests): fail fast when a reindex run ends in a terminal non-success status - #31127
Merged
Merged
Conversation
…ss status
waitForLatestRunStatus asserted containsText("Success") directly, so a run that
had already settled on "Failed" kept getting re-polled for the whole timeout —
60 minutes in external mode. In the nightly Java IT run on 2.0 that turned one
failed reindex into a job-level hang: SelectiveFieldReindexUIIT burned 3888s
before reporting, the ui-it job then hit its own 2h cap, and search-it and
scale-it were cancelled behind it and inherited a server mid-reindex. The build
spent about six hours to report a failure the badge had shown in seconds.
Wait for any terminal status first, then assert it is the expected one. Same
assertion, same message, seconds instead of an hour. Terminal set mirrors
ReindexHelpers.TERMINAL_STATUSES; badge text is upperFirst(AppRunRecord.status).
Code Review ✅ ApprovedUpdates SearchIndexAppPage to wait for any terminal reindex status before asserting success, eliminating unnecessary polling when runs fail. No issues found. OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source |
This was referenced Aug 6, 2026
k-anshul
pushed a commit
to k-anshul/OpenMetadata
that referenced
this pull request
Aug 12, 2026
…ss status (open-metadata#31127) waitForLatestRunStatus asserted containsText("Success") directly, so a run that had already settled on "Failed" kept getting re-polled for the whole timeout — 60 minutes in external mode. In the nightly Java IT run on 2.0 that turned one failed reindex into a job-level hang: SelectiveFieldReindexUIIT burned 3888s before reporting, the ui-it job then hit its own 2h cap, and search-it and scale-it were cancelled behind it and inherited a server mid-reindex. The build spent about six hours to report a failure the badge had shown in seconds. Wait for any terminal status first, then assert it is the expected one. Same assertion, same message, seconds instead of an hour. Terminal set mirrors ReindexHelpers.TERMINAL_STATUSES; badge text is upperFirst(AppRunRecord.status).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to #31125 (1.13). Test-harness only — no product code.
What breaks
SearchIndexAppPage.waitForLatestRunStatusassertedcontainsText("Success")directly. Playwright keeps re-evaluating that until it matches or the timeout expires — so a run that had already settled on the terminalFailedkept getting polled for the whole budget: 60 minutes in external mode.On
2.0that turned a single failed reindex into a whole-run outage. From run 31060569969:3594 polls against a badge that was never going to change. The
ui-itjob then hit its own 2h cap, andsearch-itandscale-itwere cancelled behind it and inherited a server mid-reindex — the "fresh runs are accepted but fail within seconds with an empty failureContext" window thatReindexHelpersalready documents. Roughly six hours to report something the badge showed in seconds.The change
Wait for any terminal status first, then assert it is the expected one. Same assertion, same failure message, seconds instead of an hour. The terminal set mirrors
ReindexHelpers.TERMINAL_STATUSES; badge text isupperFirst(AppRunRecord.status).Scope
This is the amplifier, not the cause. The underlying reindex failure on
main+2.0is #30364 un-gating the staged chunk recreate, so every full reindex now runspreflightEmbedding()-> BedrockInvokeModel-> 403 on a cluster whose node role has nobedrock:InvokeModel. That is fixed separately by pointing the Java IT deployment at DJL (in-process) embeddings. This PR just makes the failure legible when it does happen.Test plan
workflow_dispatchofk8s-java-it.ymlafter the DJL change lands;SelectiveFieldReindexUIITshould either pass or fail in seconds, never in ~65 min.🤖 Generated with Claude Code